revisions of ObsFcstAna postprocessing package #111

gmao-rreichle · 2025-06-12T12:29:29Z

Minor fixes to python package for post-processing ObsFcstAna output.

gmao-rreichle · 2025-06-12T12:38:05Z

@gmao-qliu, I started a new PR for the ObsFcstAna package because I forgot to ask you about a couple things before merging #87:

"domain" was specified in exp_sup1, but there was no check on consistency with "domain" of exp_main. I think that as long as the two experiments are in the same tile space, cross-masking would work even when the domains are different. But I'm not sure if this general functionality would have worked, and it certainly wasn't documented. It's probably simplest to require that the "domain" is the same across all experiments, and to specify "domain" only for exp_main. I took a stab at implementing this in the first commit of the PR. Please double check.
Re.

GEOSldas_GridComp/GEOSldas_App/util/postproc/ObsFcstAna_stats/Plot_stats_maps.py

Line 141 in a629ef2

grid_data, uy, ux = remap_1d_to_2d(tile_data, lat_1d = tc['com_lat'], lon_1d = tc['com_lon'])

Does this work for a cube-sphere tile space? For the cs tile space, the obs are effectively on an unstructured grid (wherever the com_lat/lon of the tiles happen to be), and the 2d array would probably require huge x and y dimensions (effectively N_tile-by-N_tile), which may crash the node. We haven't had much experience with plotting obs space maps when the model is in cs-space. I think we probably need the true "tile2grid()" functionality, which isn't what remap_1d_to_2d() provides. Perhaps I'm misreading the code. I suspect this needs further work.

amfox37 · 2025-06-15T18:35:55Z

I've been using this package quite extensively over the last few days and it seems great for my use cases at least.
The only question I have is whether the Plot_stats_timeseries.py should combine all species into a single mean by default
i.e. OmFm = np.nansum(OmFm*Nobsm)/Nobsm
As this then means the spatial_stats file is a little inconsistent with the temporal_stats file which has output by species.

gmao-qliu · 2025-06-16T19:18:04Z

I've been using this package quite extensively over the last few days and it seems great for my use cases at least. The only question I have is whether the Plot_stats_timeseries.py should combine all species into a single mean by default i.e. OmFm = np.nansum(OmFm*Nobsm)/Nobsm As this then means the spatial_stats file is a little inconsistent with the temporal_stats file which has output by species.

I can see that one might want to save stats of individual species and only combine all species before plotting. I modified to enable it.

gmao-qliu · 2025-06-16T19:19:26Z

@gmao-rreichle @amfox37 please review the changes made.

…sistent variable name for endianness, vertical alignment (read_GEOSldas.py)

…_time for clarity and consistency; vertical alignment (Plot_stats_timeseries.py, Plot_stats_maps.py)

GEOSldas_App/util/shared/python/tile2grid.py

…, Plot_stats_maps.py): - removed Nmin from calc_temporal_stats_from_sums() - moved read of stats file to Plot_stats_maps.py - edited/added comments - fixed vertical alignment

…a_stats.py)

gmao-rreichle

@gmao-qliu : The main scripts are still rather confusing. I added a few commits and comments but ran out of time to really wrap my mind around the whole thing. In a nutshell, I think we need to clearly separate the tasks accomplished by the various "main" scripts. If we have plot functions, they should focus on plotting and avoid doing things like averaging metrics across species or applying the Nmin threshold. The latter would be better off in the main() portion of the respective scripts. Not sure if all of the comments taken together make sense. We can discuss in person tomorrow

GEOSldas_App/util/postproc/ObsFcstAna_stats/Plot_stats_maps.py

GEOSldas_App/util/postproc/ObsFcstAna_stats/Get_ObsFcstAna_stats.py

gmao-qliu · 2025-06-23T20:40:36Z

I've completed the latest round of changes. Please review and test. @gmao-rreichle @amfox37

…ons; edited comments (Plot_stats_maps.py, tile_to_latlon.py)

…s (tile_to_latlongrid.py, Plot_stats_maps.py)

…proc_ObsFcstAna.py, Plot_stats_*.py)

gmao-rreichle

@gmao-qliu : I made a few more changes (https://github.com/GEOS-ESM/GEOSldas_GridComp/compare/f5fa08c..b4db565?w=1; white-space changes hidden) and added a few comments below.

GEOSldas_App/util/postproc/ObsFcstAna_stats/Get_ObsFcstAna_sums.py

GEOSldas_App/util/postproc/ObsFcstAna_stats/Plot_stats_maps.py

GEOSldas_App/util/shared/python/tile_to_latlongrid.py

GEOSldas_App/util/shared/python/tile2grid.py

…ile_to_latlongrid.py)

GEOSldas_App/util/shared/python/tile_to_latlongrid.py

cleaned up "domain" variables in ObsFcstAna postprocessing package

a629ef2

gmao-rreichle assigned gmao-qliu Jun 12, 2025

gmao-rreichle added 0-diff post-processing labels Jun 12, 2025

gmao-qliu added 7 commits June 16, 2025 10:16

add read_tilegrids() function

9267abb

update tile2grid options

60c3019

add tilegrid info. in config

0b81664

add tilegrids info

33b9123

change remap_1d_to_2d() to tile2grid()

79ff6a1

correct module name for tile2grid

b706b90

change to save stats of individual species

861923b

gmao-rreichle and others added 6 commits June 17, 2025 08:36

Merge branch 'develop' into feature/rreichle/postproc_ObsFcstAna_fixes

546733d

cleanup: corrected comments, removed redundant import statements, con…

57bae3c

…sistent variable name for endianness, vertical alignment (read_GEOSldas.py)

removed "qliu" from sample output path (user_config.py)

25270fe

fixed vertical alignment (postproc_ObsFcstAna.py)

2a3d6e7

replaced variable name start/end/current_month with start/end/current…

2423fb3

…_time for clarity and consistency; vertical alignment (Plot_stats_timeseries.py, Plot_stats_maps.py)

clarify saved stats variables

80a68eb

gmao-rreichle commented Jun 17, 2025

View reviewed changes

GEOSldas_App/util/shared/python/tile2grid.py Outdated Show resolved Hide resolved

gmao-rreichle added 3 commits June 17, 2025 15:59

minor changes in processing of temporal stats (postproc_ObsFcstAna.py…

6237a89

…, Plot_stats_maps.py): - removed Nmin from calc_temporal_stats_from_sums() - moved read of stats file to Plot_stats_maps.py - edited/added comments - fixed vertical alignment

fixed vertical alignment (Plot_stats_timeseries.py)

a285d6a

dummy white-space change to facilitate Github comments (Get_ObsFcstAn…

8df8196

…a_stats.py)

gmao-rreichle commented Jun 17, 2025

View reviewed changes

Merge branch 'develop' into feature/rreichle/postproc_ObsFcstAna_fixes

719297b

gmao-rreichle changed the title ~~minor fixes for ObsFcstAna postprocessing package~~ revisions of ObsFcstAna postprocessing package Jun 18, 2025

gmao-rreichle and others added 2 commits June 18, 2025 14:45

Updated CHANGELOG.md

4c9755d

rename file and remove plotting options to avoid confusion

5160c6c

gmao-qliu added 3 commits June 23, 2025 16:14

minor changes

76ccea9

minor correction

caa0f90

remove old file

f5fa08c

gmao-rreichle added 5 commits June 24, 2025 08:46

corrected intro comments (Get_ObsFcstAna_sums.py)

dc78876

revised computation of plotting grid spacing; removed obsolete functi…

387834b

…ons; edited comments (Plot_stats_maps.py, tile_to_latlon.py)

renamed tile_to_latlon.py -> tile_to_latlongrid.py for clarity

abdeace

add tile2grid.py; revised computation of spatial avg of temporal stat…

693ff65

…s (tile_to_latlongrid.py, Plot_stats_maps.py)

minor fixes to comments, variable names, and vertical alignment (post…

b4db565

…proc_ObsFcstAna.py, Plot_stats_*.py)

gmao-rreichle commented Jun 24, 2025

View reviewed changes

gmao-rreichle mentioned this pull request Jun 24, 2025

Is GEOSldas_LDAS.rc ignored? #119

Closed

gmao-qliu and others added 6 commits June 24, 2025 16:57

add missing return line and minor edits

33f2fca

change max to mean tile values for grid

88739fa

reverse comment change

bc8e44b

nodata_tol -> nodata_tol_frac

f97f620

fix imports, minor modification

e07721c

minimal edits of comments; white-space changes (Plot_stats_maps.py, t…

351e0a0

…ile_to_latlongrid.py)

gmao-rreichle commented Jun 24, 2025

View reviewed changes

GEOSldas_App/util/shared/python/tile_to_latlongrid.py Outdated Show resolved Hide resolved

gmao-rreichle added 0-diff trivial very, very obvious 0-diff change and removed 0-diff labels Jun 24, 2025

gmao-qliu and others added 5 commits June 25, 2025 09:30

correction for missing files

a48ac51

change loop order

6293a9f

tg bug fix and typos

34bd3e7

quiet divide by nan

b0b3ed1

Merge branch 'develop' into feature/rreichle/postproc_ObsFcstAna_fixes

145de6c

gmao-rreichle marked this pull request as ready for review June 25, 2025 20:17

gmao-rreichle requested a review from a team as a code owner June 25, 2025 20:17

weiyuan-jiang approved these changes Jun 25, 2025

View reviewed changes

gmao-rreichle merged commit c287bc0 into develop Jun 25, 2025
8 checks passed

gmao-rreichle deleted the feature/rreichle/postproc_ObsFcstAna_fixes branch June 25, 2025 20:55

revisions of ObsFcstAna postprocessing package #111

revisions of ObsFcstAna postprocessing package #111

Uh oh!

Conversation

gmao-rreichle commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gmao-rreichle commented Jun 12, 2025

Uh oh!

amfox37 commented Jun 15, 2025

Uh oh!

gmao-qliu commented Jun 16, 2025

Uh oh!

gmao-qliu commented Jun 16, 2025

Uh oh!

Uh oh!

gmao-rreichle left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gmao-qliu commented Jun 23, 2025

Uh oh!

gmao-rreichle left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gmao-rreichle commented Jun 12, 2025 •

edited

Loading